Tracking surgical pin

ABSTRACT

An example surgical pin configured to be installed in a bone of a patient includes a distal end; a shaft comprising a plurality of longitudinally spaced visually marked regions separated by non-marked regions; and a proximal end.

This patent application claims the benefit of U.S. Provisional PatentApplication No. 62/940,819, filed Nov. 26, 2019 and U.S. ProvisionalPatent Application No. 62/940,826, filed Nov. 26, 2019, the entirety ofeach of which is incorporated by reference herein.

BACKGROUND

Surgical joint repair procedures involve repair and/or replacement of adamaged or diseased joint. Many times, a surgical joint repairprocedure, such as joint arthroplasty as an example, involves replacingthe damaged joint with a prosthetic, or set of prosthetics, that isimplanted into the patient's bone. To assist with positioning, thesurgical procedure often involves the use of surgical instruments tocontrol the shaping of the surface of the damaged bone and cutting ordrilling of bone to accept the prosthetic. The use of some surgicalinstruments (e.g., cannulated instruments) involves the use of surgicalpins installed into bone.

SUMMARY

In some orthopedic surgical procedures, a surgeon may implant one ormore implant devices in a patient. The surgeon may perform varioussurgical steps to prepare the patient's bone to receive the implantdevice. These surgical steps may include insertion of guide pins,modifications to a surface of the bone (e.g., via reaming), removal ofportions of the bone (e.g., resection), creating anchorage points, orother surgical steps.

A visualization device may display virtual guidance that assists asurgeon in performing one or more of the surgical steps to prepare thepatient's bone to receive the implant device. For instance, thevisualization device may display a virtual axis to indicate a physicalaxis along which the surgeon is to install a surgical pin in a bone of apatient. The virtual axis may correspond to a planned orientation and aplanned position of the surgical pin. The surgeon may achieve correctperformance of the surgical step by aligning a shaft of the surgical pinwith the displayed virtual axis, activating a driver of the surgicalpin, and advancing the shaft of the surgical pin along the displayedvirtual axis. However, in some scenarios, a surgeon may not install asurgical pin correctly. For instance, due to various issues, the surgeonmay install the pin at an incorrect orientation.

In accordance with one or more techniques of this disclosure, avisualization device may provide virtual guidance to assist a surgeon incorrecting the installation of a surgical pin. For instance, afterinitial installation of a surgical pin into a bone of a patient, thevisualization device may determine whether an actual orientation of thesurgical pin (e.g., as installed) matches a planned orientation of thesurgical pin. If the actual orientation does not match the plannedorientation, the visualization device may output virtual guidance toassist a surgeon in correcting the installation of a surgical pin.

In some scenarios, it may be difficult for the visualization device tobe able to determine the actual orientation of a traditional surgicalpin. For instance, where the visualization device is worn on a head ofthe surgeon who is looking down at a surgical field, it may be difficultto determine the orientation of a surgical pin in the surgical fieldwhere the surgical pin is a solid piece of metal.

In accordance with one or more techniques of this disclosure, a surgicalpin may include one or more visually marked regions configured tofacilitate detection of the surgical pin. For instance, a surgical pinmay include two etched or otherwise visually differentiated regionsalong a main body. A visualization device may utilize the visuallymarked regions to determine an orientation and/or a position of thesurgical pin. For instance, the visualization device may utilize one ormore cameras to capture an image of the surgical field that includes thesurgical pin. The visualization device may analyze the image to identifyend points of each of the one or more visually marked regions, anddetermine a three-dimensional (3D) location of each end point. Based onthe 3D locations of the end points, the visualization device maydetermine the orientation and/or the position of the surgical pin. Inthis way, the techniques of this disclosure enable automatedidentification of surgical pins.

The details of various examples of the disclosure are set forth in theaccompanying drawings and the description below. Various features,objects, and advantages will be apparent from the description, drawings,and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an orthopedic surgical system according toan example of this disclosure.

FIG. 2 is a block diagram of an orthopedic surgical system that includesa mixed reality (MR) system, according to an example of this disclosure.

FIG. 3 is a flowchart illustrating example phases of a surgicallifecycle.

FIG. 4 is a flowchart illustrating preoperative, intraoperative andpostoperative workflows in support of an orthopedic surgical procedure.

FIG. 5 is a schematic representation of a visualization device for usein a mixed reality (MR) system, according to an example of thisdisclosure.

FIG. 6 is a block diagram illustrating example components of avisualization device for use in a mixed reality (MR) system, accordingto an example of this disclosure.

FIGS. 7 and 8 are conceptual diagrams illustrating an MR systemproviding virtual guidance for installation of a guide pin in a bone, inaccordance with one or more techniques of this disclosure.

FIGS. 9 and 10 are conceptual diagrams illustrating an MR systemproviding virtual guidance for installation of a guide pin in a bone, inaccordance with one or more techniques of this disclosure.

FIG. 11 is a conceptual diagram illustrating a pin installed at anincorrect orientation.

FIGS. 12A-12E are conceptual diagrams of virtual guidance that may bedisplayed to assist a surgeon in correcting installation of a surgicalpin, in accordance with one or more techniques of this disclosure.

FIG. 13 is a flowchart illustrating example techniques for diagnosingand correcting the installation of surgical pins, in accordance with oneor more techniques of this disclosure.

FIG. 14 is a conceptual diagram of surgical pin 1400 that includes oneor more visually marked regions configured to facilitate detection ofthe surgical pin, in accordance with one or more techniques of thisdisclosure.

FIG. 15 is a flowchart illustrating example techniques for tracking theposition and/or orientation of a surgical pin, in accordance with one ormore techniques of this disclosure.

DETAILED DESCRIPTION

Orthopedic surgery can involve implanting one or more implant devices torepair or replace a patient's damaged or diseased joint. Virtualsurgical planning tools that use image data of the diseased or damagedjoint may be used to generate an accurate three-dimensional bone modelthat can be viewed and manipulated preoperatively by the surgeon. Thesetools can enhance surgical outcomes by allowing the surgeon to simulatethe surgery, select or design an implant that more closely matches thecontours of the patient's actual bone, and select or design surgicalinstruments and guide tools that are adapted specifically for repairingthe bone of a particular patient. Use of these planning tools typicallyresults in generation of a preoperative surgical plan, complete with animplant and surgical instruments that are selected or manufactured forthe individual patient.

A surgeon may want to view details of the preoperative surgical planrelative to the patient's real bone during the actual procedure in orderto more efficiently and accurately position and orient the implantcomponents. For example, the surgeon may want to obtain intraoperativevisualization that provides guidance for positioning and orientation ofimplant components, guidance for preparation of bone or tissue toreceive the implant components, guidance for reviewing the details of aprocedure or procedural step, and/or guidance for selection of tools orimplants and tracking of surgical procedure workflow.

Accordingly, this disclosure describes systems and methods for using amixed reality (MR) visualization system to assist with creation,implementation, verification, and/or modification of a surgical planbefore and during a surgical procedure. Because MR may be used tointeract with the surgical plan, this disclosure may also refer to thesurgical plan as a “virtual” surgical plan. Visualization tools otherthan or in addition to mixed reality visualization systems may be usedin accordance with techniques of this disclosure. A surgical plan, e.g.,as generated by the BLUEPRINT™ system, available from Wright MedicalGroup, N.V., or another surgical planning platform, may includeinformation defining a variety of features of a surgical procedure, suchas features of particular surgical procedure steps to be performed on apatient by a surgeon according to the surgical plan including, forexample, bone or tissue preparation steps and/or steps for selection,modification and/or placement of implant components. Such informationmay include, in various examples, dimensions, shapes, angles, surfacecontours, and/or orientations of implant components to be selected ormodified by surgeons, dimensions, shapes, angles, surface contoursand/or orientations to be defined in bone or tissue by the surgeon inbone or tissue preparation steps, and/or positions, axes, planes, angleand/or entry points defining placement of implant components by thesurgeon relative to patient bone or tissue. Information such asdimensions, shapes, angles, surface contours, and/or orientations ofanatomical features of the patient may be derived from imaging (e.g.,x-ray, CT, MRI, ultrasound or other images), direct observation, orother techniques.

In this disclosure, the term “mixed reality” (MR) refers to thepresentation of virtual objects such that a user sees images thatinclude both real, physical objects and virtual objects. Virtual objectsmay include text, 2-dimensional surfaces, 3-dimensional models, or otheruser-perceptible elements that are not actually present in the physical,real-world environment in which they are presented as coexisting. Inaddition, virtual objects described in various examples of thisdisclosure may include graphics, images, animations or videos, e.g.,presented as 3D virtual objects or 2D virtual objects. Virtual objectsmay also be referred to as virtual elements. Such elements may or maynot be analogs of real-world objects. In some examples, in mixedreality, a camera may capture images of the real world and modify theimages to present virtual objects in the context of the real world. Insuch examples, the modified images may be displayed on a screen, whichmay be head-mounted, handheld, or otherwise viewable by a user. Thistype of mixed reality is increasingly common on smartphones, such aswhere a user can point a smartphone's camera at a sign written in aforeign language and see in the smartphone's screen a translation in theuser's own language of the sign superimposed on the sign along with therest of the scene captured by the camera. In some examples, in mixedreality, see-through (e.g., transparent) holographic lenses, which maybe referred to as waveguides, may permit the user to view real-worldobjects, i.e., actual objects in a real-world environment, such as realanatomy, through the holographic lenses and also concurrently viewvirtual objects.

The Microsoft HOLOLENS™ headset, available from Microsoft Corporation ofRedmond, Wash., is an example of a MR device that includes see-throughholographic lenses, sometimes referred to as waveguides, that permit auser to view real-world objects through the lens and concurrently viewprojected 3D holographic objects. The Microsoft HOLOLENS™ headset, orsimilar waveguide-based visualization devices, are examples of an MRvisualization device that may be used in accordance with some examplesof this disclosure. Some holographic lenses may present holographicobjects with some degree of transparency through see-through holographiclenses so that the user views real-world objects and virtual,holographic objects. In some examples, some holographic lenses may, attimes, completely prevent the user from viewing real-world objects andinstead may allow the user to view entirely virtual environments. Theterm mixed reality may also encompass scenarios where one or more usersare able to perceive one or more virtual objects generated byholographic projection. In other words, “mixed reality” may encompassthe case where a holographic projector generates holograms of elementsthat appear to a user to be present in the user's actual physicalenvironment.

In some examples, in mixed reality, the positions of some or allpresented virtual objects are related to positions of physical objectsin the real world. For example, a virtual object may be tethered to atable in the real world, such that the user can see the virtual objectwhen the user looks in the direction of the table but does not see thevirtual object when the table is not in the user's field of view. Insome examples, in mixed reality, the positions of some or all presentedvirtual objects are unrelated to positions of physical objects in thereal world. For instance, a virtual item may always appear in the topright of the user's field of vision, regardless of where the user islooking.

Augmented reality (AR) is similar to MR in the presentation of bothreal-world and virtual elements, but AR generally refers topresentations that are mostly real, with a few virtual additions to“augment” the real-world presentation. For purposes of this disclosure,MR is considered to include AR. For example, in AR, parts of the user'sphysical environment that are in shadow can be selectively brightenedwithout brightening other areas of the user's physical environment. Thisexample is also an instance of MR in that the selectively-brightenedareas may be considered virtual objects superimposed on the parts of theuser's physical environment that are in shadow.

Furthermore, in this disclosure, the term “virtual reality” (VR) refersto an immersive artificial environment that a user experiences throughsensory stimuli (such as sights and sounds) provided by a computer.Thus, in virtual reality, the user may not see any physical objects asthey exist in the real world. Video games set in imaginary worlds are acommon example of VR. The term “VR” also encompasses scenarios where theuser is presented with a fully artificial environment in which somevirtual object's locations are based on the locations of correspondingphysical objects as they relate to the user. Walk-through VR attractionsare examples of this type of VR.

The term “extended reality” (XR) is a term that encompasses a spectrumof user experiences that includes virtual reality, mixed reality,augmented reality, and other user experiences that involve thepresentation of at least some perceptible elements as existing in theuser's environment that are not present in the user's real-worldenvironment. Thus, the term “extended reality” may be considered a genusfor MR and VR. XR visualizations may be presented in any of thetechniques for presenting mixed reality discussed elsewhere in thisdisclosure or presented using techniques for presenting VR, such as VRgoggles.

Visualization tools may utilize patient image data to generatethree-dimensional models of bone contours to facilitate preoperativeplanning for joint repairs and replacements.

These tools allow surgeons to design and/or select surgical guides andimplant components that closely match the patient's anatomy. These toolscan improve surgical outcomes by customizing a surgical plan for eachpatient. An example of such a visualization tool for shoulder repairs isthe BLUEPRINT™ system available from Wright Medical Group, N.V. TheBLUEPRINT™ system provides the surgeon with two-dimensional planar viewsof the bone repair region as well as a three-dimensional virtual modelof the repair region. The surgeon can use the BLUEPRINT™ system toselect, design or modify appropriate implant components, determine howbest to position and orient the implant components and how to shape thesurface of the bone to receive the components, and design, select ormodify surgical guide tool(s) or instruments to carry out the surgicalplan. The information generated by the BLUEPRINT™ system is compiled ina preoperative surgical plan for the patient that is stored in adatabase at an appropriate location (e.g., on a server in a wide areanetwork, a local area network, or a global network) where it can beaccessed by the surgeon or other care provider, including before andduring the actual surgery.

FIG. 1 is a block diagram of an orthopedic surgical system 100 accordingto an example of this disclosure. Orthopedic surgical system 100includes a set of subsystems. In the example of FIG. 1, the subsystemsinclude a virtual planning system 102, a planning support system 104, amanufacturing and delivery system 106, an intraoperative guidance system108, a medical education system 110, a monitoring system 112, apredictive analytics system 114, and a communications network 116. Inother examples, orthopedic surgical system 100 may include more, fewer,or different subsystems. For example, orthopedic surgical system 100 mayomit medical education system 110, monitoring system 112, predictiveanalytics system 114, and/or other subsystems. In some examples,orthopedic surgical system 100 may be used for surgical tracking, inwhich case orthopedic surgical system 100 may be referred to as asurgical tracking system. In other cases, orthopedic surgical system 100may be generally referred to as a medical device system.

Users of orthopedic surgical system 100 may use virtual planning system102 to plan orthopedic surgeries. Users of orthopedic surgical system100 may use planning support system 104 to review surgical plansgenerated using orthopedic surgical system 100. Manufacturing anddelivery system 106 may assist with the manufacture and delivery ofitems needed to perform orthopedic surgeries. Intraoperative guidancesystem 108 provides guidance to assist users of orthopedic surgicalsystem 100 in performing orthopedic surgeries. Medical education system110 may assist with the education of users, such as healthcareprofessionals, patients, and other types of individuals. Pre- andpostoperative monitoring system 112 may assist with monitoring patientsbefore and after the patients undergo surgery. Predictive analyticssystem 114 may assist healthcare professionals with various types ofpredictions. For example, predictive analytics system 114 may applyartificial intelligence techniques to determine a classification of acondition of an orthopedic joint, e.g., a diagnosis, determine whichtype of surgery to perform on a patient and/or which type of implant tobe used in the procedure, determine types of items that may be neededduring the surgery, and so on.

The subsystems of orthopedic surgical system 100 (i.e., virtual planningsystem 102, planning support system 104, manufacturing and deliverysystem 106, intraoperative guidance system 108, medical education system110, pre- and postoperative monitoring system 112, and predictiveanalytics system 114) may include various systems. The systems in thesubsystems of orthopedic surgical system 100 may include various typesof computing systems, computing devices, including server computers,personal computers, tablet computers, smartphones, display devices,Internet of Things (IoT) devices, visualization devices (e.g., mixedreality (MR) visualization devices, virtual reality (VR) visualizationdevices, holographic projectors, or other devices for presentingextended reality (XR) visualizations), surgical tools, and so on. Aholographic projector, in some examples, may project a hologram forgeneral viewing by multiple users or a single user without a headset,rather than viewing only by a user wearing a headset. For example,virtual planning system 102 may include a MR visualization device andone or more server devices, planning support system 104 may include oneor more personal computers and one or more server devices, and so on. Acomputing system is a set of one or more computing systems configured tooperate as a system. In some examples, one or more devices may be sharedbetween two or more of the subsystems of orthopedic surgical system 100.For instance, in the previous examples, virtual planning system 102 andplanning support system 104 may include the same server devices.

In the example of FIG. 1, the devices included in the subsystems oforthopedic surgical system 100 may communicate using communicationsnetwork 116. Communications network 116 may include various types ofcommunication networks including one or more wide-area networks, such asthe Internet, local area networks, and so on. In some examples,communications network 116 may include wired and/or wirelesscommunication links.

Many variations of orthopedic surgical system 100 are possible inaccordance with techniques of this disclosure. Such variations mayinclude more or fewer subsystems than the version of orthopedic surgicalsystem 100 shown in FIG. 1. For example, FIG. 2 is a block diagram of anorthopedic surgical system 200 that includes one or more mixed reality(MR) systems, according to an example of this disclosure. Orthopedicsurgical system 200 may be used for creating, verifying, updating,modifying and/or implementing a surgical plan. In some examples, thesurgical plan can be created preoperatively, such as by using a virtualsurgical planning system (e.g., the BLUEPRINT™ system), and thenverified, modified, updated, and viewed intraoperatively, e.g., using MRvisualization of the surgical plan. In other examples, orthopedicsurgical system 200 can be used to create the surgical plan immediatelyprior to surgery or intraoperatively, as needed. In some examples,orthopedic surgical system 200 may be used for surgical tracking, inwhich case orthopedic surgical system 200 may be referred to as asurgical tracking system. In other cases, orthopedic surgical system 200may be generally referred to as a medical device system.

In the example of FIG. 2, orthopedic surgical system 200 includes apreoperative surgical planning system 202, a healthcare facility 204(e.g., a surgical center or hospital), a storage system 206, and anetwork 208 that allows a user at healthcare facility 204 to accessstored patient information, such as medical history, image datacorresponding to the damaged joint or bone and various parameterscorresponding to a surgical plan that has been created preoperatively(as examples). Preoperative surgical planning system 202 may beequivalent to virtual planning system 102 of FIG. 1 and, in someexamples, may generally correspond to a virtual planning system similaror identical to the BLUEPRINT™ system.

In the example of FIG. 2, healthcare facility 204 includes a mixedreality (MR) system 212. In some examples of this disclosure, MR system212 includes one or more processing device(s) (P) 210 to providefunctionalities that will be described in further detail below.Processing device(s) 210 may also be referred to as processor(s). Inaddition, one or more users of MR system 212 (e.g., a surgeon, nurse, orother care provider) can use processing device(s) (P) 210 to generate arequest for a particular surgical plan or other patient information thatis transmitted to storage system 206 via network 208. In response,storage system 206 returns the requested patient information to MRsystem 212. In some examples, the users can use other processingdevice(s) to request and receive information, such as one or moreprocessing devices that are part of MR system 212, but not part of anyvisualization device, or one or more processing devices that are part ofa visualization device (e.g., visualization device 213) of MR system212, or a combination of one or more processing devices that are part ofMR system 212, but not part of any visualization device, and one or moreprocessing devices that are part of a visualization device (e.g.,visualization device 213) that is part of MR system 212.

In some examples, multiple users can simultaneously use MR system 212.For example, MR system 212 can be used in a spectator mode in whichmultiple users each use their own visualization devices so that theusers can view the same information at the same time and from the samepoint of view. In some examples, MR system 212 may be used in a mode inwhich multiple users each use their own visualization devices so thatthe users can view the same information from different points of view.Different users may be located locally or remotely relative to oneanother, while interacting within MR system 212. If one or more usersare remote, then those remote users may view similar virtual informationto that of other local users while viewing different real-world viewsthan the local users.

In some examples, processing device(s) 210 can provide a user interfaceto display data and receive input from users at healthcare facility 204.Processing device(s) 210 may be configured to control visualizationdevice 213 to present a user interface. Furthermore, processingdevice(s) 210 may be configured to control visualization device 213 topresent virtual images, such as 3D virtual models, 2D images, and so on.Processing device(s) 210 can include a variety of different processingor computing devices, such as servers, desktop computers, laptopcomputers, tablets, mobile phones and other electronic computingdevices, or processors within such devices. In some examples, one ormore of processing device(s) 210 can be located remote from healthcarefacility 204. In some examples, processing device(s) 210 reside withinvisualization device 213. In some examples, at least one of processingdevice(s) 210 is external to visualization device 213. In some examples,one or more processing device(s) 210 reside within visualization device213 and one or more of processing device(s) 210 are external tovisualization device 213.

In the example of FIG. 2, MR system 212 also includes one or more memoryor storage device(s) (M) 215 for storing data and instructions ofsoftware that can be executed by processing device(s) 210. Theinstructions of software can correspond to the functionality of MRsystem 212 described herein. In some examples, the functionalities of avirtual surgical planning application, such as the BLUEPRIN™ system, canalso be stored and executed by processing device(s) 210 in conjunctionwith memory storage device(s) (M) 215. For instance, memory or storagesystem 215 may be configured to store data corresponding to at least aportion of a virtual surgical plan. In some examples, storage system 206may be configured to store data corresponding to at least a portion of avirtual surgical plan. In some examples, memory or storage device(s) (M)215 reside within visualization device 213. In some examples, memory orstorage device(s) (M) 215 are external to visualization device 213. Insome examples, memory or storage device(s) (M) 215 include a combinationof one or more memory or storage devices within visualization device 213and one or more memory or storage devices external to the visualizationdevice.

Network 208 may be equivalent to network 116. Network 208 can includeone or more wide area networks, local area networks, and/or globalnetworks (e.g., the Internet) that connect preoperative surgicalplanning system 202 and MR system 212 to storage system 206. Storagesystem 206 can include one or more databases that can contain patientinformation, medical information, patient image data, and parametersthat define the surgical plans. For example, medical images of thepatient's diseased or damaged bone typically are generatedpreoperatively in preparation for an orthopedic surgical procedure. Themedical images can include images of the relevant bone(s) taken alongthe sagittal plane and the coronal plane of the patient's body. Themedical images can include X-ray images, magnetic resonance imaging(MRI) images, computerized tomography (CT) images, ultrasound images,and/or any other type of 2D or 3D image that provides information aboutthe relevant surgical area. Storage system 206 also can include dataidentifying the implant components selected for a particular patient(e.g., type, size, etc.), surgical guides selected for a particularpatient, and details of the surgical procedure, such as entry points,cutting planes, drilling axes, reaming depths, etc. Storage system 206can be a cloud-based storage system (as shown) or can be located athealthcare facility 204 or at the location of preoperative surgicalplanning system 202 or can be part of MR system 212 or visualizationdevice (VD) 213, as examples.

MR system 212 can be used by a surgeon before (e.g., preoperatively) orduring the surgical procedure (e.g., intraoperatively) to create,review, verify, update, modify and/or implement a surgical plan. In someexamples, MR system 212 may also be used after the surgical procedure(e.g., postoperatively) to review the results of the surgical procedure,assess whether revisions are required, or perform other postoperativetasks. To that end, MR system 212 may include a visualization device 213that may be worn by the surgeon and (as will be explained in furtherdetail below) is operable to display a variety of types of information,including a 3D virtual image of the patient's diseased, damaged, orpostsurgical joint and details of the surgical plan, such as a 3Dvirtual image of the prosthetic implant components selected for thesurgical plan, 3D virtual images of entry points for positioning theprosthetic components, alignment axes and cutting planes for aligningcutting or reaming tools to shape the bone surfaces, or drilling toolsto define one or more holes in the bone surfaces, in the surgicalprocedure to properly orient and position the prosthetic components,surgical guides and instruments and their placement on the damagedjoint, and any other information that may be useful to the surgeon toimplement the surgical plan. MR system 212 can generate images of thisinformation that are perceptible to the user of the visualization device213 before and/or during the surgical procedure.

In some examples, MR system 212 includes multiple visualization devices(e.g., multiple instances of visualization device 213) so that multipleusers can simultaneously see the same images and share the same 3Dscene. In some such examples, one of the visualization devices can bedesignated as the master device and the other visualization devices canbe designated as observers or spectators. Any observer device can bere-designated as the master device at any time, as may be desired by theusers of MR system 212.

In this way, FIG. 2 illustrates a surgical planning system that includesa preoperative surgical planning system 202 to generate a virtualsurgical plan customized to repair an anatomy of interest of aparticular patient. For example, the virtual surgical plan may include aplan for an orthopedic joint repair surgical procedure, such as one of astandard total shoulder arthroplasty or a reverse shoulder arthroplasty.In this example, details of the virtual surgical plan may includedetails relating to at least one of preparation of glenoid bone orpreparation of humeral bone. In some examples, the orthopedic jointrepair surgical procedure is one of a stemless standard total shoulderarthroplasty, a stemmed standard total shoulder arthroplasty, a stemlessreverse shoulder arthroplasty, a stemmed reverse shoulder arthroplasty,an augmented glenoid standard total shoulder arthroplasty, and anaugmented glenoid reverse shoulder arthroplasty.

The virtual surgical plan may include a 3D virtual model correspondingto the anatomy of interest of the particular patient and a 3D model of aprosthetic component matched to the particular patient to repair theanatomy of interest or selected to repair the anatomy of interest.Furthermore, in the example of FIG. 2, the surgical planning systemincludes a storage system 206 to store data corresponding to the virtualsurgical plan. The surgical planning system of FIG. 2 also includes MRsystem 212, which may comprise visualization device 213. In someexamples, visualization device 213 is wearable by a user. In someexamples, visualization device 213 is held by a user, or rests on asurface in a place accessible to the user. MR system 212 may beconfigured to present a user interface via visualization device 213. Theuser interface is visually perceptible to the user using visualizationdevice 213. For instance, in one example, a screen of visualizationdevice 213 may display real-world images and the user interface on ascreen. In some examples, visualization device 213 may project virtual,holographic images onto see-through holographic lenses and also permit auser to see real-world objects of a real-world environment through thelenses. In other words, visualization device 213 may comprise one ormore see-through holographic lenses and one or more display devices thatpresent imagery to the user via the holographic lenses to present theuser interface to the user.

In some examples, visualization device 213 is configured such that theuser can manipulate the user interface (which is visually perceptible tothe user when the user is wearing or otherwise using visualizationdevice 213) to request and view details of the virtual surgical plan forthe particular patient, including a 3D virtual model of the anatomy ofinterest (e.g., a 3D virtual bone of the anatomy of interest) and a 3Dmodel of the prosthetic component selected to repair an anatomy ofinterest. In some such examples, visualization device 213 is configuredsuch that the user can manipulate the user interface so that the usercan view the virtual surgical plan intraoperatively, including (at leastin some examples) the 3D virtual model of the anatomy of interest (e.g.,a 3D virtual bone of the anatomy of interest). In some examples, MRsystem 212 can be operated in an augmented surgery mode in which theuser can manipulate the user interface intraoperatively so that the usercan visually perceive details of the virtual surgical plan projected ina real environment, e.g., on a real anatomy of interest of theparticular patient. In this disclosure, the terms real and real worldmay be used in a similar manner. For example, MR system 212 may presentone or more virtual objects that provide guidance for preparation of abone surface and placement of a prosthetic implant on the bone surface.Visualization device 213 may present one or more virtual objects in amanner in which the virtual objects appear to be overlaid on an actual,real anatomical object of the patient, within a real-world environment,e.g., by displaying the virtual object(s) with actual, real-worldpatient anatomy viewed by the user through holographic lenses. Forexample, the virtual objects may be 3D virtual objects that appear toreside within the real-world environment with the actual, realanatomical object.

FIG. 3 is a flowchart illustrating example phases of a surgicallifecycle 300. In the example of FIG. 3, surgical lifecycle 300 beginswith a preoperative phase (302). During the preoperative phase, asurgical plan is developed. The preoperative phase may be followed by amanufacturing and delivery phase (304). During the manufacturing anddelivery phase, patient-specific items, such as parts and equipment,needed for executing the surgical plan are manufactured and delivered toa surgical site. For instance, a patient specific implant may bemanufactured based on a design generated during the preoperative phase.An intraoperative phase follows the manufacturing and delivery phase(306). The surgical plan is executed during the intraoperative phase. Inother words, one or more persons perform the surgery on the patientduring the intraoperative phase. The intraoperative phase is followed bythe postoperative phase (308). The postoperative phase includesactivities occurring after the surgical plan is complete. For example,the patient may be monitored during the postoperative phase forcomplications.

As described in this disclosure, orthopedic surgical system 100 (FIG. 1)may be used in one or more of preoperative phase 302, the manufacturingand delivery phase 304, the intraoperative phase 306, and thepostoperative phase 308. For example, virtual planning system 102 andplanning support system 104 may be used in preoperative phase 302.Manufacturing and delivery system 106 may be used in the manufacturingand delivery phase 304. Intraoperative guidance system 108 may be usedin intraoperative phase 306. Some of the systems of FIG. 1 may be usedin multiple phases of FIG. 3. For example, medical education system 110may be used in one or more of preoperative phase 302, intraoperativephase 306, and postoperative phase 308; pre- and postoperativemonitoring system 112 may be used in preoperative phase 302 andpostoperative phase 308. Predictive analytics system 114 may be used inpreoperative phase 302 and postoperative phase 308.

Various workflows may exist within the surgical process of FIG. 3. Forexample, different workflows within the surgical process of FIG. 3 maybe appropriate for different types of surgeries. FIG. 4 is a flowchartillustrating preoperative, intraoperative and postoperative workflows insupport of an orthopedic surgical procedure. In the example of FIG. 4,the surgical process begins with a medical consultation (400). Duringthe medical consultation (400), a healthcare professional evaluates amedical condition of a patient. For instance, the healthcareprofessional may consult the patient with respect to the patient'ssymptoms. During the medical consultation (400), the healthcareprofessional may also discuss various treatment options with thepatient. For instance, the healthcare professional may describe one ormore different surgeries to address the patient's symptoms.

Furthermore, the example of FIG. 4 includes a case creation step (402).In other examples, the case creation step occurs before the medicalconsultation step. During the case creation step, the medicalprofessional or other user establishes an electronic case file for thepatient. The electronic case file for the patient may includeinformation related to the patient, such as data regarding the patient'ssymptoms, patient range of motion observations, data regarding asurgical plan for the patient, medical images of the patients, notesregarding the patient, billing information regarding the patient, and soon.

The example of FIG. 4 includes a preoperative patient monitoring phase(404). During the preoperative patient monitoring phase, the patient'ssymptoms may be monitored. For example, the patient may be sufferingfrom pain associated with arthritis in the patient's shoulder. In thisexample, the patient's symptoms may not yet rise to the level ofrequiring an arthroplasty to replace the patient's shoulder. However,arthritis typically worsens over time. Accordingly, the patient'ssymptoms may be monitored to determine whether the time has come toperform a surgery on the patient's shoulder. Observations from thepreoperative patient monitoring phase may be stored in the electroniccase file for the patient. In some examples, predictive analytics system114 may be used to predict when the patient may need surgery, to predicta course of treatment to delay or avoid surgery or make otherpredictions with respect to the patient's health.

Additionally, in the example of FIG. 4, a medical image acquisition stepoccurs during the preoperative phase (406). During the image acquisitionstep, medical images of the patient are generated. The medical imagesmay be generated in a variety of ways. For instance, the images may begenerated using a Computed Tomography (CT) process, a Magnetic ResonanceImaging (MRI) process, an ultrasound process, or another imagingprocess. The medical images generated during the image acquisition stepinclude images of an anatomy of interest of the patient. For instance,if the patient's symptoms involve the patient's shoulder, medical imagesof the patient's shoulder may be generated. The medical images may beadded to the patient's electronic case file. Healthcare professionalsmay be able to use the medical images in one or more of thepreoperative, intraoperative, and postoperative phases. In someexamples, the medical images may be segmented into anatomical parts. Forinstance, medical images of the patient's shoulder may be segmented intoa scapula, a humerus, etc. Three-dimensional (3D) models of theanatomical parts may be generated.

Furthermore, in the example of FIG. 4, an automatic processing step mayoccur (408). During the automatic processing step, virtual planningsystem 102 (FIG. 1) may automatically develop a preliminary surgicalplan for the patient. In some examples of this disclosure, virtualplanning system 102 may use machine learning techniques to develop thepreliminary surgical plan based on information in the patient's virtualcase file.

The example of FIG. 4 also includes a manual correction step (410).During the manual correction step, one or more human users may check andcorrect the determinations made during the automatic processing step. Insome examples of this disclosure, one or more users may use mixedreality or virtual reality visualization devices during the manualcorrection step. In some examples, changes made during the manualcorrection step may be used as training data to refine the machinelearning techniques applied by virtual planning system 102 during theautomatic processing step.

A virtual planning step (412) may follow the manual correction step inFIG. 4. During the virtual planning step, a healthcare professional maydevelop a surgical plan for the patient. In some examples of thisdisclosure, one or more users may use mixed reality or virtual realityvisualization devices during development of the surgical plan for thepatient. As discussed in further detail below, during the virtualplanning step, virtual planning system 102 may design a patient matchedimplant.

Furthermore, in the example of FIG. 4, intraoperative guidance may begenerated (414). The intraoperative guidance may include guidance to asurgeon on how to execute the surgical plan. In some examples of thisdisclosure, virtual planning system 102 may generate at least part ofthe intraoperative guidance. In some examples, the surgeon or other usermay contribute to the intraoperative guidance.

Additionally, in the example of FIG. 4, a step of selecting andmanufacturing surgical items is performed (416). During the step ofselecting and manufacturing surgical items, manufacturing and deliverysystem 106 (FIG. 1) may manufacture surgical items for use during thesurgery described by the surgical plan. For example, the surgical itemsmay include surgical implants (e.g., generic and/or patient specific),surgical tools, and other items required to perform the surgerydescribed by the surgical plan.

In the example of FIG. 4, a surgical procedure may be performed withguidance from intraoperative system 108 (FIG. 1) (418). For example, asurgeon may perform the surgery while wearing a head-mounted MRvisualization device of intraoperative system 108 that presents guidanceinformation to the surgeon. The guidance information may help guide thesurgeon through the surgery, providing guidance for various surgicalsteps in a surgical workflow, including sequence of surgical steps,details of individual surgical steps, and tool or implant selection,implant placement and position, and bone surface preparation for varioussurgical steps in the surgical procedure workflow.

Postoperative patient monitoring may occur after completion of thesurgical procedure (420). During the postoperative patient monitoringstep, healthcare outcomes of the patient may be monitored. Healthcareoutcomes may include relief from symptoms, ranges of motion,complications, performance of implanted surgical items, and so on. Pre-and postoperative monitoring system 112 (FIG. 1) may assist in thepostoperative patient monitoring step.

The medical consultation, case creation, preoperative patientmonitoring, image acquisition, automatic processing, manual correction,and virtual planning steps of FIG. 4 are part of preoperative phase 302of FIG. 3. The surgical procedures with guidance steps of FIG. 4 is partof intraoperative phase 306 of FIG. 3. The postoperative patientmonitoring step of FIG. 4 is part of postoperative phase 308 of FIG. 3.

As mentioned above, one or more of the subsystems of orthopedic surgicalsystem 100 may include one or more mixed reality (MR) systems, such asMR system 212 (FIG. 2). Each MR system may include a visualizationdevice. For instance, in the example of FIG. 2, MR system 212 includesvisualization device 213. In some examples, in addition to including avisualization device, an MR system may include external computingresources that support the operations of the visualization device. Forinstance, the visualization device of an MR system may becommunicatively coupled to a computing device (e.g., a personalcomputer, backpack computer, smartphone, etc.) that provides theexternal computing resources. Alternatively, adequate computingresources may be provided on or within visualization device 213 toperform necessary functions of the visualization device.

FIG. 5 is a schematic representation of visualization device 213 for usein an MR system, such as MR system 212 of FIG. 2, according to anexample of this disclosure. As shown in the example of FIG. 5,visualization device 213 can include a variety of electronic componentsfound in a computing system, including one or more processor(s) 514(e.g., microprocessors or other types of processing units) and memory516 that may be mounted on or within a frame 518. Furthermore, in theexample of FIG. 5, visualization device 213 may include a transparentscreen 520 that is positioned at eye level when visualization device 213is worn by a user. In some examples, screen 520 can include one or moreliquid crystal displays (LCDs) or other types of display screens onwhich images are perceptible to a surgeon who is wearing or otherwiseusing visualization device 213 via screen 520. Other display examplesinclude organic light emitting diode (OLED) displays. In some examples,visualization device 213 can operate to project 3D images onto theuser's retinas using techniques known in the art.

In some examples, screen 520 includes see-through holographic lenses,sometimes referred to as waveguides, that permit a user to seereal-world objects through (e.g., beyond) the lenses and also seeholographic imagery projected into the lenses and onto the user'sretinas by displays, such as liquid crystal on silicon (LCoS) displaydevices, which are sometimes referred to as light engines or projectors,operating as an example of a holographic projection system 538 withinvisualization device 213. In other words, visualization device 213 mayinclude one or more see-through holographic lenses to present virtualimages to a user. Hence, in some examples, visualization device 213 canoperate to project 3D images onto the user's retinas via screen 520,e.g., formed by holographic lenses. In this manner, visualization device213 may be configured to present a 3D virtual image to a user within areal-world view observed through screen 520, e.g., such that the virtualimage appears to form part of the real-world environment. In someexamples, visualization device 213 may be a Microsoft HOLOLENS™ headset,available from Microsoft Corporation, of Redmond, Wash., USA, or asimilar device, such as, for example, a similar MR visualization devicethat includes waveguides. The HOLOLENS™ device can be used to present 3Dvirtual objects via holographic lenses, or waveguides, while permittinga user to view actual objects in a real-world scene, i.e., in areal-world environment, through the holographic lenses.

Although the example of FIG. 5 illustrates visualization device 213 as ahead-wearable device, visualization device 213 may have other forms andform factors. For instance, in some examples, visualization device 213may be a handheld smartphone or tablet.

Visualization device 213 can also generate a virtual user interface (UI)522 that is visible to the user, e.g., as holographic imagery projectedinto see-through holographic lenses as described above. For example, UI522 can include a variety of selectable virtual widgets 524 that allowthe user to interact with a mixed reality (MR) system, such as MR system212 of FIG. 2. Imagery presented by visualization device 213 mayinclude, for example, one or more 3D virtual objects. Details of anexample of UI 522 are described elsewhere in this disclosure.Visualization device 213 also can include a speaker or other sensorydevices 526 that may be positioned adjacent the user's ears. Sensorydevices 526 can convey audible information or other perceptibleinformation (e.g., vibrations) to assist the user of visualizationdevice 213.

Visualization device 213 can also include a transceiver 528 to connectvisualization device 213 to a processing device 510 and/or to network208 and/or to a computing cloud, such as via a wired communicationprotocol or a wireless protocol, e.g., Wi-Fi, Bluetooth, etc.Visualization device 213 also includes a variety of sensors to collectsensor data, such as one or more optical camera(s) 530 (or other opticalsensors) and one or more depth camera(s) 532 (or other depth sensors),mounted to, on or within frame 518. In some examples, the opticalsensor(s) 530 are operable to scan the geometry of the physicalenvironment in which a user of MR system 212 is located (e.g., anoperating room) and collect two-dimensional (2D) optical image data(either monochrome or color). Depth sensor(s) 532 are operable toprovide 3D image data, such as by employing time of flight, stereo orother known or future-developed techniques for determining depth andthereby generating image data in three dimensions. Other sensors caninclude motion sensors 533 (e.g., Inertial Mass Unit (IMU) sensors,accelerometers, etc.) to assist with tracking movement.

MR system 212 processes the sensor data so that geometric,environmental, textural, or other types of landmarks (e.g., corners,edges or other lines, walls, floors, objects) in the user's environmentor “scene” can be defined and movements within the scene can bedetected. As an example, the various types of sensor data can becombined or fused so that the user of visualization device 213 canperceive 3D images that can be positioned, or fixed and/or moved withinthe scene. When a 3D image is fixed in the scene, the user can walkaround the 3D image, view the 3D image from different perspectives, andmanipulate the 3D image within the scene using hand gestures, voicecommands, gaze line (or direction) and/or other control inputs. Asanother example, the sensor data can be processed so that the user canposition a 3D virtual object (e.g., a bone model) on an observedphysical object in the scene (e.g., a surface, the patient's real bone,etc.) and/or orient the 3D virtual object with other virtual imagesdisplayed in the scene. In some examples, the sensor data can beprocessed so that the user can position and fix a virtual representationof the surgical plan (or other widget, image or information) onto asurface, such as a wall of the operating room. Yet further, in someexamples, the sensor data can be used to recognize surgical instrumentsand the position and/or location of those instruments.

Visualization device 213 may include one or more processors 514 andmemory 516, e.g., within frame 518 of the visualization device. In someexamples, one or more external computing resources 536 process and storeinformation, such as sensor data, instead of or in addition to in-frameprocessor(s) 514 and memory 516. In this way, data processing andstorage may be performed by one or more processors 514 and memory 516within visualization device 213 and/or some of the processing andstorage requirements may be offloaded from visualization device 213.Hence, in some examples, one or more processors that control theoperation of visualization device 213 may be within visualization device213, e.g., as processor(s) 514. Alternatively, in some examples, atleast one of the processors that controls the operation of visualizationdevice 213 may be external to visualization device 213, e.g., asprocessor(s) 210. Likewise, operation of visualization device 213 may,in some examples, be controlled in part by a combination one or moreprocessors 514 within the visualization device and one or moreprocessors 210 external to visualization device 213.

For instance, in some examples, when visualization device 213 is in thecontext of FIG. 2, processing of the sensor data can be performed byprocessing device(s) 210 in conjunction with memory or storage device(s)(M) 215. In some examples, processor(s) 514 and memory 516 mounted toframe 518 may provide sufficient computing resources to process thesensor data collected by cameras 530, 532 and motion sensors 533. Insome examples, the sensor data can be processed using a SimultaneousLocalization and Mapping (SLAM) algorithm, or other known orfuture-developed algorithms for processing and mapping 2D and 3D imagedata and tracking the position of visualization device 213 in the 3Dscene. In some examples, image tracking may be performed using sensorprocessing and tracking functionality provided by the MicrosoftHOLOLENS™ system, e.g., by one or more sensors and processors 514 withina visualization device 213 substantially conforming to the MicrosoftHOLOLENS™ device or a similar mixed reality (MR) visualization device.

In some examples, MR system 212 can also include user-operated controldevice(s) 534 that allow the user to operate MR system 212, use MRsystem 212 in spectator mode (either as master or observer), interactwith UI 522 and/or otherwise provide commands or requests to processingdevice(s) 210 or other systems connected to network 208. As examples,control device(s) 534 can include a microphone, a touch pad, a controlpanel, a motion sensor or other types of control input devices withwhich the user can interact.

FIG. 6 is a block diagram illustrating example components ofvisualization device 213 for use in a MR system. In the example of FIG.6, visualization device 213 includes processors 514, a power supply 600,display device(s) 602, speakers 604, microphone(s) 606, input device(s)608, output device(s) 610, storage device(s) 612, sensor(s) 614, andcommunication devices 616. In the example of FIG. 6, sensor(s) 616 mayinclude depth sensor(s) 532, optical sensor(s) 530, motion sensor(s)533, and orientation sensor(s) 618. Optical sensor(s) 530 may includecameras, such as Red-Green-Blue (RGB) video cameras, infrared cameras,or other types of sensors that form images from light. Display device(s)602 may display imagery to present a user interface to the user.

Speakers 604, in some examples, may form part of sensory devices 526shown in FIG. 5. In some examples, display devices 602 may includescreen 520 shown in FIG. 5. For example, as discussed with reference toFIG. 5, display device(s) 602 may include see-through holographiclenses, in combination with projectors, that permit a user to seereal-world objects, in a real-world environment, through the lenses, andalso see virtual 3D holographic imagery projected into the lenses andonto the user's retinas, e.g., by a holographic projection system. Inthis example, virtual 3D holographic objects may appear to be placedwithin the real-world environment. In some examples, display devices 602include one or more display screens, such as LCD display screens, OLEDdisplay screens, and so on. The user interface may present virtualimages of details of the virtual surgical plan for a particular patient.

In some examples, a user may interact with and control visualizationdevice 213 in a variety of ways. For example, microphones 606, andassociated speech recognition processing circuitry or software, mayrecognize voice commands spoken by the user and, in response, performany of a variety of operations, such as selection, activation, ordeactivation of various functions associated with surgical planning,intra-operative guidance, or the like. As another example, one or morecameras or other optical sensors 530 of sensors 614 may detect andinterpret gestures (such as hand motions, hand gestures, finger motions,finger gestures, eye blinks, or other physical gestures) in order toperform operations as described above. As a further example, sensors 614may sense gaze direction and perform various operations as describedelsewhere in this disclosure. In some examples, input devices 608 mayreceive manual input from a user, e.g., via a handheld controllerincluding one or more buttons, a keypad, a touchscreen, joystick,trackball, and/or other manual input media, and perform, in response tothe manual user input, various operations as described above.

As discussed above, surgical lifecycle 300 may include a preoperativephase 302 (FIG. 3). One or more users may use orthopedic surgical system100 in preoperative phase 302. For instance, orthopedic surgical system100 may include virtual planning system 102 to help the one or moreusers generate a virtual surgical plan that may be customized to ananatomy of interest of a particular patient. As described herein, thevirtual surgical plan may include a 3-dimensional virtual model thatcorresponds to the anatomy of interest of the particular patient and a3-dimensional model of one or more prosthetic components matched to theparticular patient to repair the anatomy of interest or selected torepair the anatomy of interest. The virtual surgical plan also mayinclude a 3-dimensional virtual model of guidance information to guide asurgeon in performing the surgical procedure, e.g., in preparing bonesurfaces or tissue and placing implantable prosthetic hardware relativeto such bone surfaces or tissue.

A visualization system, such as MR visualization system 212, may beconfigured to display virtual guidance including one or more virtualguides for performing work on a portion of a patient's anatomy. In someexamples, a user such as a surgeon may view real-world objects in areal-world scene. The real-world scene may be in a real-worldenvironment such as a surgical operating room. In this disclosure, theterms real and real-world may be used in a similar manner. Thereal-world objects viewed by the user in the real-world scene mayinclude the patient's actual, real anatomy, such as an actual glenoid orhumerus, exposed during surgery. The user may view the real-worldobjects via a see-through (e.g., transparent) screen, such assee-through holographic lenses, of a head-mounted MR visualizationdevice, such as visualization device 213, and also see virtual guidancesuch as virtual MR objects that appear to be projected on the screen orwithin the real-world scene, such that the MR guidance object(s) appearto be part of the real-world scene, e.g., with the virtual objectsappearing to the user to be integrated with the actual, real-worldscene. For example, the virtual guidance may be projected on the screenof a MR visualization device, such as visualization device 213, suchthat the virtual guidance is overlaid on, and appears to be placedwithin, an actual, observed view of the patient's actual bone viewed bythe surgeon through the transparent screen, e.g., through see-throughholographic lenses. Hence, in this example, the virtual guidance may bea virtual 3D object that appears to be part of the real-worldenvironment, along with actual, real-world objects.

A screen through which the surgeon views the actual, real anatomy andalso observes the virtual objects, such as virtual anatomy and/orvirtual surgical guidance, may include one or more see-throughholographic lenses. The holographic lenses, sometimes referred to as“waveguides,” may permit the user to view real-world objects through thelenses and display projected holographic objects for viewing by theuser. As discussed above, an example of a suitable head-mounted MRdevice for visualization device 213 is the Microsoft HOLOLENS™ headset,available from Microsoft Corporation, of Redmond, Wash., USA. TheHOLOLENS™ headset includes see-through, holographic lenses, alsoreferred to as waveguides, in which projected images are presented to auser. The HOLOLENS™ headset also includes an internal computer, camerasand sensors, and a projection system to project the holographic contentvia the holographic lenses for viewing by the user. In general, theMicrosoft HOLOLENS™ headset or a similar MR visualization device mayinclude, as mentioned above, LCoS display devices that project imagesinto holographic lenses, also referred to as waveguides, e.g., viaoptical components that couple light from the display devices to opticalwaveguides. The waveguides may permit a user to view a real-world scenethrough the waveguides while also viewing a 3D virtual image presentedto the user via the waveguides. In some examples, the waveguides may bediffraction waveguides.

The visualization system (e.g., MR system 212/visualization device 213)may be configured to display different types of virtual guidance.Examples of virtual guidance include, but are not limited to, a virtualpoint, a virtual axis, a virtual angle, a virtual path, a virtual plane,virtual reticle, and a virtual surface or contour. As discussed above,the visualization system (e.g., MR system 212/visualization device 213)may enable a user to directly view the patient's anatomy via a lens bywhich the virtual guides are displayed, e.g., projected. The virtualguidance may guide or assist various aspects of the surgery. Forinstance, a virtual guide may guide at least one of preparation ofanatomy for attachment of the prosthetic or attachment of the prostheticto the anatomy.

The visualization system may obtain parameters for the virtual guidesfrom a virtual surgical plan, such as the virtual surgical plandescribed herein. Example parameters for the virtual guides include, butare not necessarily limited to, guide location, guide orientation, guidetype, guide color, etc.

The visualization system may display a virtual guide in a manner inwhich the virtual guide appears to be overlaid on an actual, realobject, within a real-world environment, e.g., by displaying the virtualguide(s) with actual, real-world objects (e.g., at least a portion ofthe patient's anatomy) viewed by the user through holographic lenses.For example, the virtual guidance may be 3D virtual objects that appearto reside within the real-world environment with the actual, realobject.

The techniques of this disclosure are described below with respect to ashoulder arthroplasty surgical procedure. Examples of shoulderarthroplasties include, but are not limited to, reversed arthroplasty,augmented reverse arthroplasty, standard total shoulder arthroplasty,augmented total shoulder arthroplasty, and hemiarthroplasty. However,the techniques are not so limited, and the visualization system may beused to provide virtual guidance information, including virtual guidesin any type of surgical procedure. Other example procedures in which avisualization system, such as MR system 212, may be used to providevirtual guidance include, but are not limited to, other types oforthopedic surgeries; any type of procedure with the suffix “plasty,”“stomy,” “ectomy,” “clasia,” or “centesis,”; orthopedic surgeries forother joints, such as elbow, wrist, finger, hip, knee, ankle or toe, orany other orthopedic surgical procedure in which precision guidance isdesirable. For instance, a visualization system may be used to providevirtual guidance for an ankle arthroplasty surgical procedure.

As discussed above, a MR system (e.g., MR system 212, MR system 1800A ofFIG. 18, etc.) may receive a virtual surgical plan for attaching animplant to a patient and/or preparing bones, soft tissue or otheranatomy of the patient to receive the implant. The virtual surgical planmay specify various surgical steps to be performed and variousparameters for the surgical steps to be performed. As one example, thevirtual surgical plan may specify a location on the patient's bone(e.g., glenoid, humerus, tibia, talus, etc.) for attachment of a guidepin. As another example, the virtual surgical plan may specify locationsand/or orientations of one or more anchorage locations (e.g., screws,stems, pegs, keels, etc.).

FIGS. 7 and 8 are conceptual diagrams illustrating an MR systemproviding virtual guidance for installation of a guide pin in a bone, inaccordance with one or more techniques of this disclosure. In FIGS. 7and 8 and other FIGS., for purposes of illustration, some of thesurrounding tissue and some bone has been omitted for ease ofillustration. As shown in FIG. 7, MR system 212 may display virtual axis3400 on or relative to humeral head 3204 of humerus 3200. FIG. 7 andsubsequent figures illustrate one example of what the surgeon, or otheruser, would see when viewing via visualization device 213. Inparticular, when viewing via visualization device 213 from the viewshown in FIG. 7, the surgeon may see a portion of humerus 3200 andvirtual axis 3400 (and/or other virtual guidance) overlaid on theportion of humerus 3200.

To display virtual axis 3400, MR system 212 may determine a location ona virtual model of humerus 3200 at which a guide is to be installed. MRsystem 212 may obtain the location from a virtual surgical plan (e.g.,the virtual surgical plan described above as generated by virtualplanning system 202). The location obtained by MR system 212 may specifyone or both of coordinates of a point on the virtual model and a vector(e.g., a planned position and a planned orientation). The point may bethe position at which the guide is to be installed and the vector mayindicate the angle/slope at which the guide is to be installed. As such,MR system 212 may display a virtual drilling axis having parametersobtained from the virtual surgical plan, and the virtual drilling axismay be configured to guide drilling of one or more holes in the glenoid(e.g., for attachment of a guide pin to the scapula).

A virtual model of humerus 3200 may be registered with humerus 3200 suchthat coordinates on the virtual model approximately correspond tocoordinates on humerus 3200. For instance, MR system 212 may generate atransformation matrix between the virtual model of humerus 3200 and anobserved portion of humerus 3200. This transformation matrix may allowfor translation along the x, y, and z axes of the virtual model androtation about the x, y and z axes in order to achieve and maintainalignment between the virtual and observed bones. In some examples,after registration is complete, MR system 212 utilizes the results ofthe registration to perform simultaneous localization and mapping (SLAM)(or any other tracking algorithm) to maintain alignment of the virtualmodel to the corresponding observed object. As such, by displayingvirtual axis 3400 at the determined location on the virtual model, MRsystem 212 may display virtual axis 3400 at the planned position onhumerus 3200.

The surgeon may attach a guide pin (e.g., a surgical pin) to humerus3200 using the displayed virtual guidance. For instance, where the guidepin includes a self-tapping threaded distal tip, the surgeon may alignthe guide pin with the displayed virtual axis 3400 and utilize a drillor other instrument to install the guide pin in humerus 3200.

FIG. 8 is a conceptual diagram illustrating guide 3500 as installed inhumeral head 3204. Guide 3500 may take the form of an elongated pin tobe mounted in a hole formed in the humeral head. As shown in FIGS. 7 and8, by displaying virtual axis 3400, a surgeon may install guide 3500 atthe planned position on humeral head 3204.

As discussed above, FIG. 7 illustrates an example of what the surgeon,or other user, would see when viewing via visualization device 213 fromthe view shown in FIG. 7. In particular, FIG. 7 shows what the surgeonwould see when the surgeon's gaze line is from a side view/substantiallyorthogonal to the axis of the surgical step being performed (e.g.,virtual axis 3400). However, the surgeon is not likely to view thepatient from such an angle when operating a driver of a rotating tool(e.g., a drill or motor that rotates the guide pin (e.g., guide 3500), adrill bit, a reamer, or the like). Instead, when operating the driver ofthe rotating tool, the surgeon is likely to view the patient from behindthe drill or motor while operating the drill or motor, with a gaze linesubstantially parallel to an axis of the surgical step being performed.

FIGS. 9 and 10 are conceptual diagrams illustrating an MR systemproviding virtual guidance for installation of a guide pin in a bone, inaccordance with one or more techniques of this disclosure. FIGS. 9 and10 are similar to FIGS. 7 and 8 in that they depict virtual guidance forinstallation of a guide pin in a bone. However, where FIGS. 7 and 8depict virtual guidance for installation of a guide pin in a humerus,FIGS. 9 and 10 depict virtual guidance for installation of a guide pinin a scapula. In particular, as shown in FIG. 9, MR system 212 maydisplay virtual guidance, e.g., in the form of virtual axis 5104, onglenoid 5102 of scapula 5100. To display virtual axis 5104, MR system212 may determine a location on a virtual model of glenoid 5102 at whicha guide is to be installed. MR system 212 may obtain the location from avirtual surgical plan (e.g., the virtual surgical plan described above).The location obtained by MR system 212 may specify one or both ofcoordinates of a point on the virtual model and a vector (e.g., aplanned position and a planned orientation). The point may be theposition at which the guide is to be installed (e.g., the plannedposition) and the vector may indicate the angle/slope at which the guideis to be installed (e.g., the planned orientation). As such, MR system212 may display a virtual reaming axis having parameters (e.g.,position, size, and/or orientation relative to the virtual model of thescapula) obtained from the virtual surgical plan. The displayed virtualreaming axis may be configured to guide reaming of the glenoid.

The surgeon may attach a physical guide using the displayed virtualguidance. As one example, where the guide is a guide pin with aself-tapping threaded distal tip, the surgeon may align the guide pinwith the displayed virtual axis 5104 and utilize a drill or otherinstrument to install the guide pin. As another example, where the guideis a guide pin without a self-tapping tip, the surgeon may align a drillbit of a drill with the displayed virtual axis 5104 and operate thedrill to form a hole to receive the guide pin and then install the guidepin in the hole. In some examples, MR system 212 may display depthguidance information to enable the surgeon to install the guide pin to aplanned depth.

FIG. 10 is a conceptual diagram illustrating guide 5200, i.e., a guidepin in this example, as installed in glenoid 5102. As shown in FIGS. 9and 10, by displaying virtual axis 5104, a surgeon may drill inalignment with the virtual axis, which may be referred to as a reamingaxis, and thereby form a hole for installation of guide 5200 at theplanned position on glenoid 5102. In this way, MR system 212 may enablethe installation of a guide without the need for an additionalmechanical guide.

As discussed above, in some examples, the surgeon may not install asurgical pin (e.g., a guide pin) correctly. For instance, the surgeonmay install guide pin 5200 in glenoid 5102 at an incorrect orientation.In some examples, the surgeon may install guide pin 5200 at an incorrectorientation because the virtual axis may be at least partially occludedby the drill, therefore making it difficult to maintain alignmentbetween guide pin 5200 and the virtual axis.

FIG. 11 is a conceptual diagram illustrating a pin installed at anincorrect orientation. As shown in FIG. 11, guide 5200 is actuallyinstalled in glenoid 5102 at an orientation that is different than aplanned orientation. In particular, while FIG. 11 illustrates guide 5200installed at the correct location on glenoid 5102, the actualorientation of guide 5200 is approximately 15 degrees off from theplanned orientation denoted by virtual axis 5104.

In accordance with one or more techniques of this disclosure, MR system212 may automatically determine whether a surgical pin was installedcorrectly. For instance, after initial installation of a surgical pin(e.g., guide 5200) into a bone of a patient (e.g., glenoid 5102), MRsystem 212 may determine whether an actual orientation of the surgicalpin (e.g., as installed) matches a planned orientation of the surgicalpin. Further details of how MR system 212 may determine whether theactual orientation matches the planned orientation are described below.Additionally or alternatively, the surgeon may visually determinewhether the actual orientation of the surgical pin matches the plannedorientation and provide user input (e.g., push a button, provide averbal command, etc.) to MR system 212 indicating whether the actualorientation of the surgical pin matches the planned orientation.

If the actual orientation does match the planned orientation, MR system212 may proceed to provide guidance for subsequent steps of the surgicalprocedure. For instance, MR system 212 may provide guidance to utilize acannulated tool that is guided by the surgical pin.

If the actual orientation does not match the planned orientation, MRsystem 212 may output However, in some examples, MR system 212 mayprovide virtual guidance to assist the surgeon in correcting theinstallation of a surgical pin (e.g., with or without also providing anoutput indicating that the actual orientation does not match the plannedorientation). In both examples, the surgeon may correct the installationof the surgical pin via any sufficient technique. For instance, thesurgeon may correct the orientation of the surgical pin by utilizingbending pliers to physically bend the material of the surgical pin.

FIGS. 12A-12E are conceptual diagrams of virtual guidance that may bedisplayed to assist a surgeon in correcting installation of a surgicalpin, in accordance with one or more techniques of this disclosure. Asshown in FIG. 12A, to output the virtual guidance to assist the surgeonin correcting the installation of guide 5200, MR system 212 may displayvirtual axis 5104 (i.e., a virtual axis corresponding to the plannedorientation). The surgeon may utilize the displayed virtual guidance tocorrect the orientation of guide 5200. For instance, as shown in FIG.12A, the surgeon may place bending pliers 1200 (or any other suitabletool) at a base of guide 5200. As shown in FIG. 12B, the surgeon maythen align a bending axis of bending pliers 1200 with a desired plane ofbending. For instance, the surgeon may rotate bending pliers 1200 suchthat activation of bending pliers 1200 will result in guide 5200 beingbent toward the planned orientation (e.g., the orientation of virtualaxis 5104).

As shown in FIG. 12C, the surgeon may then activate bending pliers 1200(e.g., squeeze handles of bending pliers 1200) to bend guide 5200towards the planned orientation. As shown in FIG. 12D, the surgeon maycontinue to use bending pliers 1200 until guide 5200 reaches the plannedorientation. With guide 5200 bent to the planned orientation, thesurgeon may remove bending pliers 1200. As shown in FIG. 12E, afterbending, the actual orientation of guide 5200 corresponds to the plannedorientation. As also shown in FIG. 12E the bending may result in acrease or other deformation in the shape of guide 5200 (e.g., there willbe a bend in guide 5200 at the point of bending). In this way, MR system212 may guide a surgeon in correcting installation of a guide pin.

In some examples, MR system 212 may periodically determine whether asurgical pin (e.g., guide 5200) was installed as planned. Where MRsystem 212 determines that the surgical pin was not originally installedcorrectly (e.g., determines that the surgical pin was not installed asplanned at a first time), MR system 212 may provide virtual guidance toassist the surgeon in correcting the installation as discussed above andmay periodically determine whether installation of the surgical pin hasbeen corrected. For instance, as the surgeon performs steps to correctinstallation of the surgical pin, MR system 212 may periodicallydetermine a current actual orientation of the surgical pin and comparethe determined current actual orientation with the planned orientationto determine whether installation of the guide pin has been corrected.

MR system 212 may determine that installation of the guide pin has beencorrected in response to determining that the current actual orientationof the surgical pin matches the planned orientation (e.g., is within atolerable range, where exact match may be possible but is notnecessary). Responsive to determining that installation of the surgicalpin has been corrected (e.g., responsive to determining that thesurgical pin is installed as planned at a second time that is after thefirst time), MR system 212 may output an indication that the surgicalpin is now installed correctly. MR system 212 may output the indicationusing any suitable channel. For instance, MR system 212 may output anycombination of visual, audible, or haptic indications that the surgicalpin is now installed correctly. As one example, where the virtualguidance to assist the surgeon in correcting the installation of thesurgical pin includes a virtual axis corresponding to the plannedorientation, MR system 212 may adjust a visual characteristic of thedisplayed virtual axis (e.g., change a color, such as turning from redto green) responsive to determining that installation of the surgicalpin has been corrected. As another example, responsive to determiningthat installation of the surgical pin has been corrected, MR system 212may display text (e.g., cause visualization device 213 to display text)indicating that installation of the surgical pin has been corrected.

While described above as displaying a virtual axis, MR system 212 maydisplay any variety of virtual guidance elements to assist the surgeonin correcting the installation of the surgical pin. Example virtualguidance elements include, but are not limited to, virtual points,virtual axes, virtual angles, virtual paths, virtual planes, virtualreticles, and virtual surfaces or contours. As one example, the virtualguidance elements to assist the surgeon in correcting the installationof the surgical pin may include a virtual axis corresponding to theactual orientation of the surgical pin. For instance, MR system 212 maydisplay the virtual axis corresponding to the actual orientation of thesurgical pin and an animation of the virtual axis corresponding to theactual orientation of the surgical pin moving to the planned orientation(e.g., an animation of the surgical pin moving from the actualorientation to the planned orientation).

While described above as being used to diagnose and correct theinstallation of a surgical pin in a glenoid of a scapula, the techniquesof this disclosure are equally applicable to diagnosing and correctingthe installation of surgical pins in any anatomy of a patient. Forinstance, the techniques of this disclosure may be used to diagnose andcorrect the installation of a surgical pin in a scapula, a humerus, atibia, and/or a talus.

In any case, once the installation of the surgical pin has beencorrected, the surgeon may continue with the surgical procedure. Forinstance, the surgeon may utilize the guide pin to guide use of one ormore cannulated tools.

FIG. 13 is a flowchart illustrating example techniques for diagnosingand correcting the installation of surgical pins, in accordance with oneor more techniques of this disclosure. For purposes of explanation, thetechniques of FIG. 13 are described as being performed by MR system 212of FIG. 1. However, other mixed-reality systems may perform thetechniques of FIG. 13.

MR system 212 may obtain a planned orientation of a surgical pin (1302).For instance, MR system 212 may obtain, from a virtual surgical plan(e.g., the virtual surgical plan described above), a vector indicatingan angle/slope at which the surgical is to be installed (e.g., theplanned orientation). The vector may be obtained as relative to avirtual model of an anatomy in-which the surgical pin is to beinstalled. For instance, in the example of FIG. 10, MR system 212 mayobtain the planned orientation guide 5200 relative to a virtual model ofglenoid 5102.

MR system 212 may determine an actual orientation of the surgical pin(1304). For instance, MR system 212 may process data obtained via one ormore sensors of visualization device 213 to determine the actualorientation of the surgical pin.

MR system 212 may determine whether the surgical pin is installedcorrectly (1306). For instance, MR system 212 may compare the actualorientation of the surgical pin with the planned orientation of thesurgical pin. If a difference between the actual orientation and theplanned orientation is greater than a threshold difference (e.g., 2degrees, 5 degrees, 10 degrees, etc.), MR system 212 may determine thatthe surgical pin is not installed correctly. If the difference betweenthe actual orientation and the planned orientation is not greater thanthe threshold difference, MR system 212 may determine that the surgicalpin is installed correctly.

Responsive to determining that the surgical pin was installed correctly,MR system 212 may output an indication that the surgical pin wasinstalled correctly (“Yes” branch of 1306, 1308). For instance,visualization device 213 may display text indicating that the surgicalpin is installed correctly.

Responsive to determining that the surgical pin was not installedcorrectly, MR system 212 may output virtual guidance to assist incorrecting installation of the surgical pin (“No” branch of 1306, 1310).For instance, visualization device 213 may output any of the virtualguidance discussed above with reference to FIGS. 12A-12E.

MR system 212 may update the determination of the current orientation ofthe surgical pin (1304), and determine whether installation of thesurgical pin has been corrected based on the updated current orientation(1306). In this way, MR system 212 may diagnose and correct theinstallation of a surgical pin.

As discussed above, in some examples, it may be desirable to determine aposition and/or an orientation of a surgical pin. For instance, whendetermining whether a surgical pin was installed correctly, it may bedesirable for MR system 212 may be able to determine the position and/ororientation of the surgical pin. In some scenarios, it may be difficultfor MR system 212 to be able to determine the actual orientation and/orposition of a traditional surgical pin. For instance, wherevisualization device 213 of MR system 212 is worn on a head of thesurgeon who is looking down at a surgical field, it may be difficult forMR system 212 to determine the orientation of a surgical pin in thesurgical field.

In accordance with one or more techniques of this disclosure, a surgicalpin may include one or more visually marked regions configured tofacilitate detection of the surgical pin. For instance, a surgical pinmay include two etched or otherwise visually differentiated regionsalong a shaft. MR system 212 may utilize the visually marked regions ofthe surgical pin to determine an orientation and/or a position of thesurgical pin. For instance, visualization device 213 may utilize one ormore cameras to capture an image of the surgical field that includes thesurgical pin. MR system 212 may analyze the image to identify end pointsof each of the one or more visually marked regions, and determine athree-dimensional (3D) location of each end point. Based on the 3Dlocations of the end points, MR system 212 may determine the orientationand/or the position of the surgical pin.

FIG. 14 is a conceptual diagram of surgical pin 1400 that includes oneor more visually marked regions configured to facilitate detection ofthe surgical pin, in accordance with one or more techniques of thisdisclosure. Surgical pin 1400 may be considered to be an example ofguide 3500 of FIG. 8, or guide 5200 of FIGS. 10, 11, and 12A-12E.Examples of surgical pin 1400 include, but are not necessarily limitedto, Steinmann pins and k-wires.

As shown in FIG. 14, surgical pin 1400 includes shaft 1406 having distalend 1402 and proximal end 1404. Distal end 1402 may be configured todrill into material. For instance, distal end 1402 may have a trocarshape suitable for drilling into bone. Proximal end 1404 may beconfigured to be attached to a chuck (e.g., of a surgical motor) fortransferring rotational energy to surgical pin 1400.

As discussed above, a surgical pin may include one or more visuallymarked regions configured to facilitate detection of the surgical pin.As shown in FIG. 14, surgical pin 1400 may include visually markedregions 1408A and 1408B (collectively, “visually marked regions 1408”).Visually marked regions 1408 may be longitudinally spaced alonglongitudinal axis 1416 of surgical pin 1400. For instance, visuallymarked region 1408A is shown as being longitudinally displaced from(i.e., at a different position along longitudinal axis 1416 than)visually marked region 1408B. Visually marked regions 1408 may haveequal longitudinal length or may have different longitudinal length. Asone example, the longitudinal length of visually marked region 1408A LAmay be the same as the longitudinal length of visually marked region1408B LB. As another example, the longitudinal length of visually markedregion 1408A LA may be different than the longitudinal length ofvisually marked region 1408B LB. Visually marked regions 1408 may be inthe shape of bands or rings. In any case, visualization device 213 maystore (e.g., in memory) values representing the longitudinal lengths ofvisually marked regions 1408.

Visually marked regions 1408 may be interspersed/separated by non-markedregions 1414A-1414C (collectively, “non-marked regions 1414”). As shownin the example of FIG. 14, the marked and non-marked regions may be asfollows, from proximal end 1404 to distal end 1402, non-marked region1414A, marked region 1408A, non-marked region 1414B, marked region1408B, and non-marked region 1414C.

The boundary between a visually marked region and a non-marked regionmay define a point, which may be referred to as an end point of avisually marked region. For instance, the boundary between non-markedregion 1414A and visually marked region 1408A may define endpoint 1410A,the boundary between visually marked region 1408A and non-marked region1414B may define endpoint 1412A, the boundary between non-marked region1414B and visually marked region 1408B may define endpoint 1410B, theboundary between visually marked region 1408B and non-marked region1414C may define endpoint 1412B.

Visually marked regions 1408 may be fabricated to be visually distinctfrom non-marked regions 1414. As one example, visually marked regions1408 may have a different color than non-marked regions 1414. Forinstance, visually marked regions 1408 may be black or dark-gray whilenon-marked regions 1414 are light gray or white (e.g., metallic color).As another example, visually marked regions 1408 may be fabricated tohave a high contrast relative to non-marked regions 1414. For instance,visually marked regions 1408 may be fabricated to be darker thannon-marked regions 1414.

Visually marked regions 1408 may be fabricated using any suitableprocess. As one example, visually marked regions 1408 may be fabricatedvia etching (e.g., laser etching) surgical pin 1400. As another example,visually marked regions 1408 may be fabricated by applying paint or dyeto surgical pin 1400. Visually marked regions 1408 may be fabricatedsuch that surgical pin 1400 may be sterilized (e.g., placed in anautoclave) without visually marked regions 1408 incurring damage.

In addition to being visually distinct from non-marked regions, variousaspects of visually marked regions 1408 may be selected to facilitatethe detection of surgical pin 1400. As one example, the longitudinallengths of visually marked regions 1408 may be selected to facilitatethe detection of surgical pin 1400. For instance, to increase the visualsignal provided by the locations of the end points of visually markedregions 1408 (i.e., endpoints 1410 and 1412), the longitudinal lengthsof visually marked regions 1408 may be selected to be significantlylarger than a diameter D of surgical pin 1400. For example, LA and LBmay be selected to be at least five times D. As one specific example,where D is 2.5 mm, LA and LB may be selected to be 40 mm.

FIG. 15 is a flowchart illustrating example techniques for tracking theposition and/or orientation of a surgical pin, in accordance with one ormore techniques of this disclosure. For purposes of explanation, thetechniques of FIG. 15 are described as being performed by MR system 212of FIG. 1. However, other mixed-reality systems may perform thetechniques of FIG. 15.

MR system 212 may obtain an image of a surgical field that depicts asurgical pin (1502). For instance, one or more cameras of visualizationdevice 213 may capture an image that includes surgical pin 1400 of FIG.14, which includes one or more longitudinally spaced visually markedregions. As one example, one or more RGB cameras of visualization device213 may capture the image (e.g., a texture image of the surgical field).As another example, one or more depth cameras of visualization device213 may capture the image (e.g., may capture a depth map of the surgicalfield).

MR system 212 may identify, in the image, locations of end points ofeach of the one or more marked regions (1504). For instance, processors210 of MR system 212 may process the image (e.g., using thresholding orany other boundary detection algorithm) to identify locations of endpoints 1410A, 1412A, 1410B, and 1412B of surgical pin 1400 in the image.The identification of the end points in the image may be in the form ofwhich pixels or samples in the image correspond to the end points.

MR system 212 may determine, based on the identified locations of theend points in the image, a respective three-dimensional (3D) location ofeach respective end points of the end points of the one or more markedregions (1506). For instance, processors 210 may calculate a 3Dcoordinate (e.g., an x,y,z coordinate set) for each of end points 1410A,1412A, 1410B, and 1412B of surgical pin 1400. As one example, processors210 may determine, use a depth camera, depth values for pointscorresponding to the identified end points of each of the one or moremarked regions. As another example, processors 210 may utilize aperspective-n-point algorithm (e.g., where N=2) to determine the 3Dlocations of the end points.

In some examples, MR system 212 may determine the 3D locations based onone or more pre-determined parameters of the surgical pin. For instance,processors 210 may obtain, from memory 215 of MR system 212,longitudinal lengths of each of the marked regions of the surgical pin(e.g., obtain values for LA and LB). Processors 210 may determine therespective 3D location of each respective end point based on thelongitudinal lengths of each of the marked regions.

In some examples, MR system 212 may determine the 3D locations based onone or more pre-determined parameters of the camera(s) that captured theimage. For instance, processors 210 may obtain, from memory 215 of MRsystem 212, one or more parameters of RGB cameras of visualizationdevice 213. Processors 210 may determine the respective 3D location ofeach respective end point based on the parameters of the RGB cameras.For instance, processors 10 may transform the respective locations ofthe end points in the image to the respective 3D locations the endpoints based on pre-determined characteristics of the RGB cameras.

As discussed above, in some examples, obtained image may be a textureimage of the surgical field. In some examples, in addition to thetexture image, MR system 212 may obtain a depth map of the surgicalfield (e.g., as captured by one or more depth cameras of visualizationdevice 213). MR system 212 may, in some examples, determine the 3Dlocations based on the identified locations of the end point in theimage and the depth map. For instance, MR system 212 may map therespective locations of the end points in the texture image tocorresponding locations in the depth map (e.g., select a sample in thedepth map that corresponds to a sample in the texture image identifiedas a location of an end point). MR system 212 may determine therespective 3D locations of the end points based on the correspondinglocations in the depth map.

MR system 212 may determine, based on the determined 3D locations, aposition and/or an orientation of the surgical pin (1508). For instance,processors 210 may generate a vector connecting the 3D locations of theend points, the vector representing the orientation of the surgical pin.Processors 210 may determine the position of the surgical point as apoint at which the vector intersects with a virtual model of anatomyregistered to a corresponding portion of actual anatomy (e.g., a virtualmodel of glenoid 5102 that is registered that the patient's actualglenoid).

MR system 212 may utilize the determined position and/or orientation ofthe surgical pin for any suitable purpose. As one example, MR system 212may utilize the determined position and/or orientation of the surgicalpin to provide virtual guidance. As another example, 212 may utilize thedetermined position and/or orientation of the surgical pin to determinewhether the surgical pin was properly installed (e.g., as discussedabove).

The following numbered examples may illustrate one or more aspects ofthe disclosure:

Example 1. A surgical pin configured to be installed in a bone of apatient, the surgical pin comprising: a distal end; a shaft comprising aplurality of longitudinally spaced visually marked regions separated bynon-marked regions; and proximal end.

Example 2. The surgical pin of example 1, wherein each marked region ofthe plurality of marked regions comprises a band.

Example 3. The surgical pin of example 2, wherein a longitudinal lengthof a particular band is greater than five times a diameter of theparticular band.

Example 4. The surgical pin of any of examples 1-3, wherein theplurality of marked regions are of equal longitudinal length.

Example 5. The surgical pin of any of examples 1-4, wherein a diameterof the marked regions of the shaft is not greater than a diameter of thenon-marked regions of the shaft.

Example 6. The surgical pin of example 5, wherein the diameter of themarked regions of the shaft and the diameter of the non-marked regionsof the shaft are equal.

Example 7. The surgical pin of example 5, wherein the diameter of themarked regions of the shaft and the diameter of the non-marked regionsof the shaft are different.

Example 8. The surgical pin of example 7, wherein the diameter of themarked regions of the shaft is less than the diameter of the non-markedregions of the shaft.

Example 9. The surgical pin of example 8, wherein the diameter of themarked regions of the shaft is 2 millimeters and the diameter of thenon-marked regions of the shaft is 2.5 millimeters.

Example 10. The surgical pin of any of examples 1-9, wherein each markedregion of the plurality of marked region is etched into the shaft.

Example 11. The surgical pin of any of examples 1-10, wherein the markedregions have a different color than the non-marked regions.

Example 12. The surgical pin of any of examples 1-11, wherein the markedregions are darker than the non-marked regions.

Example 13. The surgical pin of any of examples 1-12, wherein theplurality of marked regions includes exactly two marked regions of equallongitudinal length separated by a single non-marked region.

Example 14. The surgical pin of any of examples 1-13, wherein thesurgical pin is configured to undergo sterilization, and wherein theplurality of marked regions are configured to undergo the sterilizationwithout incurring damage to the surgical pin.

Example 15. The surgical pin of any of examples 1-14, wherein thesurgical pin is a Steinmann pin.

Example 16. The surgical pin of any of examples 1-15, wherein the distalend is configured to drill into the bone, and wherein the proximal endis configured to be attached to a chuck for rotation.

Example 17. A method comprising: obtaining, via one or more cameras of avisualization device, an image of a surgical field that depicts asurgical pin that includes one or more longitudinally spaced visuallymarked regions; identifying, by one or more processors and in the image,locations of end points of each of the one or more marked regions;determining, by the one or more processors and based on the identifiedlocations of the end points in the image, a respective three-dimensional(3D) location of each respective end points of the end points of the oneor more marked regions; and determining, by the one or more processorsand based on the determined 3D locations, a position and an orientationof the surgical pin within the surgical field.

Example 18. The method of example 17, wherein obtaining the imagecomprises: obtaining, via one or more RGB cameras of the visualizationdevice, the image.

Example 19. The method of example 17 or 18, wherein determining therespective 3D location of each respective end point comprises: obtaininga pre-determined longitudinal length of the one or more marked regions;determining, based on the identified locations of the end points in theimage and the longitudinal length of the one or more marked regions, therespective 3D location of each respective end point.

Example 20. The method of any of examples 18 or 19, wherein determiningthe respective 3D location of each respective end point comprises:transforming, based on pre-determined characteristics of the one or morecameras, the respective locations of the end points in the image to therespective 3D locations the end points.

Example 21. The method of any of examples 17-20, wherein the imagecomprises a texture image, the method further comprising: obtaining, viaone or more depth cameras of the visualization device, a depth map ofthe surgical field.

Example 22. The method of example 21, wherein determining the respective3D location of each respective end point comprises: mapping therespective locations of the end points in the texture image tocorresponding locations in the depth map; and determining the respective3D locations of the end points based on the corresponding locations inthe depth map.

Example 23. The method of example 17, wherein obtaining the imagecomprises: obtaining, via one or more depth cameras of the visualizationdevice, the image.

Example 24. The method of any of examples 17-26, further comprising:displaying, via the visualization device and based on one or both of theposition and the orientation of the surgical pin, virtual guidance.

Example 25. The method of any of examples 17-27, wherein the image iscaptured during performance of a surgical procedure.

Example 26. The method of any of examples 17-27, wherein the surgicalpin comprises the surgical pin of any of examples 1-16.

Example 27. A system comprising: one or more processors that areimplemented in circuitry; and a computer-readable storage medium storinginstructions that, when executed, cause the one or more processors toperform the method of any combination of examples 17-26.

Example 28. A computer-readable storage medium storing instructionsthat, when executed, cause one or more processors to perform the methodof any combination of examples 17-26.

Example 29. A method comprising: determining, by the one or moreprocessors, an actual orientation of a surgical pin as installed in abone of a patient; obtaining, by the one or more processors, a plannedorientation of the surgical pin; determining, by the one or moreprocessors and based on a comparison between the actual orientation ofthe surgical pin and the planned orientation of the surgical pin,whether the surgical pin was installed as planned; and responsive todetermining that the surgical pin was not installed as planned,outputting, via a visualization device, virtual guidance to assist asurgeon in correcting the installation of the surgical pin.

Example 30. The method of example 29, wherein outputting the virtualguidance comprises: displaying, via the visualization device, a virtualaxis corresponding to the planned orientation.

Example 31. The method of any of examples 29 or 30, wherein determiningwhether the surgical pin was installed as planned comprises periodicallydetermining whether the surgical pin was installed as planned,determining that the surgical pin was not installed as planned comprisesdetermining that the surgical pin was not installed as planned at afirst time, the method further comprising: responsive to determiningthat the surgical pin is installed as planned at a second time that isafter the first time, outputting, via the visualization device, anindication that the surgical pin is installed correctly.

Example 32. The method of example 31, wherein outputting the indicationthat the surgical pin is installed correctly comprises outputting avisual, audible, or haptic indication that the surgical pin is installedcorrectly.

Example 33. The method of example 31 or 32, wherein outputting theindication that the surgical pin is installed correctly comprisesadjusting a visual characteristic of the displayed virtual axiscorresponding to the planned orientation.

Example 34. The method of any of examples 29-33, wherein outputting thevirtual guidance comprises: displaying, via the visualization device, avirtual axis corresponding to the actual orientation of the surgicalpin.

Example 35. The method of any of examples 29-34, wherein outputting thevirtual guidance comprises: displaying, via the visualization device, ananimation of the surgical pin moving from the actual orientation to theplanned orientation.

Example 36. The method of any of examples 29-35, wherein the bone of thepatient comprises a scapula, a humerus, a tibia, and/or a talus.

Example 37. The method of any of examples 29-36, wherein the surgicalpin comprises the surgical pin of any of examples 1-16.

Example 38. A system comprising: one or more processors that areimplemented in circuitry; and a computer-readable storage medium storinginstructions that, when executed, cause the one or more processors toperform the method of any combination of examples 29-37.

Example 39. A computer-readable storage medium storing instructionsthat, when executed, cause one or more processors to perform the methodof any combination of examples 29-37.

Example 40. Any combination of examples 1-39.

While the techniques been disclosed with respect to a limited number ofexamples, those skilled in the art, having the benefit of thisdisclosure, will appreciate numerous modifications and variations therefrom. For instance, it is contemplated that any reasonable combinationof the described examples may be performed. It is intended that theappended claims cover such modifications and variations as fall withinthe true spirit and scope of the invention.

It is to be recognized that depending on the example, certain acts orevents of any of the techniques described herein can be performed in adifferent sequence, may be added, merged, or left out altogether (e.g.,not all described acts or events are necessary for the practice of thetechniques). Moreover, in certain examples, acts or events may beperformed concurrently, e.g., through multi-threaded processing,interrupt processing, or multiple processors, rather than sequentially.

In one or more examples, the functions described may be implemented inhardware, software, firmware, or any combination thereof. If implementedin software, the functions may be stored on or transmitted over as oneor more instructions or code on a computer-readable medium and executedby a hardware-based processing unit. Computer-readable media may includecomputer-readable storage media, which corresponds to a tangible mediumsuch as data storage media, or communication media including any mediumthat facilitates transfer of a computer program from one place toanother, e.g., according to a communication protocol. In this manner,computer-readable media generally may correspond to (1) tangiblecomputer-readable storage media which is non-transitory or (2) acommunication medium such as a signal or carrier wave. Data storagemedia may be any available media that can be accessed by one or morecomputers or one or more processors to retrieve instructions, codeand/or data structures for implementation of the techniques described inthis disclosure. A computer program product may include acomputer-readable medium.

By way of example, and not limitation, such computer-readable storagemedia can comprise RAM, ROM, EEPROM, CD-ROM or other optical diskstorage, magnetic disk storage, or other magnetic storage devices, flashmemory, or any other medium that can be used to store desired programcode in the form of instructions or data structures and that can beaccessed by a computer. Also, any connection is properly termed acomputer-readable medium. For example, if instructions are transmittedfrom a website, server, or other remote source using a coaxial cable,fiber optic cable, twisted pair, digital subscriber line (DSL), orwireless technologies such as infrared, radio, and microwave, then thecoaxial cable, fiber optic cable, twisted pair, DSL, or wirelesstechnologies such as infrared, radio, and microwave are included in thedefinition of medium. It should be understood, however, thatcomputer-readable storage media and data storage media do not includeconnections, carrier waves, signals, or other transitory media, but areinstead directed to non-transitory, tangible storage media. Disk anddisc, as used herein, includes compact disc (CD), laser disc, opticaldisc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wheredisks usually reproduce data magnetically, while discs reproduce dataoptically with lasers. Combinations of the above should also be includedwithin the scope of computer-readable media.

Operations described in this disclosure may be performed by one or moreprocessors, which may be implemented as fixed-function processingcircuits, programmable circuits, or combinations thereof, such as one ormore digital signal processors (DSPs), general purpose microprocessors,application specific integrated circuits (ASICs), field programmablegate arrays (FPGAs), or other equivalent integrated or discrete logiccircuitry. Fixed-function circuits refer to circuits that provideparticular functionality and are preset on the operations that can beperformed. Programmable circuits refer to circuits that can programmedto perform various tasks and provide flexible functionality in theoperations that can be performed. For instance, programmable circuitsmay execute instructions specified by software or firmware that causethe programmable circuits to operate in the manner defined byinstructions of the software or firmware. Fixed-function circuits mayexecute software instructions (e.g., to receive parameters or outputparameters), but the types of operations that the fixed-functioncircuits perform are generally immutable. Accordingly, the terms“processor” and “processing circuity,” as used herein may refer to anyof the foregoing structures or any other structure suitable forimplementation of the techniques described herein.

Various examples have been described. These and other examples arewithin the scope of the following claims.

1. A surgical pin configured to be installed in a bone of a patient, thesurgical pin comprising: a distal end; a shaft comprising a plurality oflongitudinally spaced visually marked regions separated by non-markedregions; and a proximal end.
 2. The surgical pin of claim 1, whereineach marked region of the plurality of marked regions comprises a band.3. The surgical pin of claim 2, wherein a longitudinal length of aparticular band is greater than five times a diameter of the particularband.
 4. The surgical pin of claim 1, wherein the plurality of markedregions are of equal longitudinal length.
 5. The surgical pin of claim1, wherein a diameter of the marked regions of the shaft is not greaterthan a diameter of the non-marked regions of the shaft.
 6. The surgicalpin of claim 5, wherein the diameter of the marked regions of the shaftand the diameter of the non-marked regions of the shaft are equal. 7.The surgical pin of claim 5, wherein the diameter of the marked regionsof the shaft and the diameter of the non-marked regions of the shaft aredifferent.
 8. The surgical pin of claim 7, wherein the diameter of themarked regions of the shaft is less than the diameter of the non-markedregions of the shaft.
 9. The surgical pin of claim 8, wherein thediameter of the marked regions of the shaft is 2 millimeters and thediameter of the non-marked regions of the shaft is 2.5 millimeters. 10.The surgical pin of claim 1, wherein each marked region of the pluralityof marked region is etched into the shaft.
 11. The surgical pin of claim1, wherein the marked regions have a different color than the non-markedregions.
 12. The surgical pin of claim 1, wherein the marked regions aredarker than the non-marked regions.
 13. The surgical pin of claim 1,wherein the plurality of marked regions includes exactly two markedregions of equal longitudinal length separated by a single non-markedregion.
 14. The surgical pin of claim 1, wherein the surgical pin isconfigured to undergo sterilization, and wherein the plurality of markedregions are configured to undergo the sterilization without incurringdamage to the surgical pin.
 15. The surgical pin of claim 1, wherein thesurgical pin is a Steinmann pin.
 16. The surgical pin of claim 1,wherein the distal end is configured to drill into the bone, and whereinthe proximal end is configured to be attached to a chuck for rotation.17. A method comprising: obtaining, via one or more cameras of avisualization device, an image of a surgical field that depicts asurgical pin that includes one or more longitudinally spaced visuallymarked regions; identifying, by one or more processors and in the image,locations of end points of each of the one or more marked regions;determining, by the one or more processors and based on the identifiedlocations of the end points in the image, a respective three-dimensional(3D) location of each respective end points of the end points of the oneor more marked regions; and determining, by the one or more processorsand based on the determined 3D locations, a position and an orientationof the surgical pin within the surgical field.
 18. The method of claim17, wherein obtaining the image comprises: obtaining, via one or moreRGB cameras of the visualization device, the image.
 19. The method ofclaim 17, wherein determining the respective 3D location of eachrespective end point comprises: obtaining a pre-determined longitudinallength of the one or more marked regions; determining, based on theidentified locations of the end points in the image and the longitudinallength of the one or more marked regions, the respective 3D location ofeach respective end point. 20-27. (canceled)
 28. A computer-readablestorage medium storing instructions that, when executed, cause one ormore processors to: obtain, via one or more cameras of a visualizationdevice, an image of a surgical field that depicts a surgical pin thatincludes one or more longitudinally spaced visually marked regions;identify, in the image, locations of end points of each of the one ormore marked regions; determine, based on the identified locations of theend points in the image, a respective three-dimensional (3D) location ofeach respective end points of the end points of the one or more markedregions; and determine, by the one or more processors and based on thedetermined 3D locations, a position and an orientation of the surgicalpin within the surgical field.